Over the past several years the benefits of 3D imaging for metrology and inspection has been recognized. Applications of 3D imaging range from microelectronics inspection to mobile robot navigation, with data rates from a few hundred points per second to video rates. Accuracy of these systems is limited, at the most fundamental level, by the underlying physics of the imaging process.
Illumination Distribution and Imaging Errors
In all 3D systems the distribution of incident illumination upon a detector is analyzed in some manner, and the corresponding temporal or spatial variations analyzed to estimate the object reflectance and/or depth. Usually, the centroid of the intensity distribution is analyzed to determine the position of a point on the object surface. As such, position dependent anomalies which corrupt the intensity distribution lead to inaccuracies which may vary from a negligible "noise" contribution to visible artifacts producing very large errors which may render the imaging system useless.
Early researchers in computer vision recognized the importance of the physics of imaging and the image formation process for scene modeling and understanding. For instance, Horn in his article entitled "Understanding Image Intensities," Artificial Intelligence (1977) pp. 201-231, considered numerous imaging phenomena, including the effect of "mutual illumination," when investigating the relationship between observed image intensity and surface characteristics and geometry.
Barrow and Tennenbaum, in their article entitled "Recovering Intrinsic Scene Characteristics from Images," Computer Vision Systems (1978) pp. 3-26, developed a model of "Intrinsic Images," which related an array of intensity values and physical surface properties like depth, reflectance, and surface orientation. They were cautious, however, to limit their model based upon assumptions about the illumination, including a limitation requiring that no secondary reflections be present. Violation of this assumption would render the model invalid because of the ambiguous interpretation of a single image.
3D Imaging and Errors Caused by Inter-Reflections
Many real-world imaging applications involve estimation of more complex geometries. Objects having such geometries and, in particular, concave shapes are susceptible to the occurrence of multiple reflections within the instantaneous field of view ("IFOV"), thus leading to ambiguous image data. The reflections of the energy by the surface which allow the imaging to take place may be either specular (direct), diffuse (scattered), or, usually, components of both. The multiplicity of image points falling on the sensor can make it impossible to determine the correct height estimate, as only one point represents the true object point of interest. The ambiguity generally leads either to an uncertain choice as to which is the correct image point or a weighted average of the various image points which lead to an incorrect estimate.
A large percentage of 3D imaging techniques now in use are based upon the triangulation principle. Several references discuss the limitations of instruments based upon triangulation, the most well known limitation being occlusion effects.
However, there are fewer studies discussing the less obvious, but perhaps even more severe, limitation associated with secondary reflections (also called intra-scene reflections, inter-reflections, mutual illumination, secondary illumination, multiple reflections, self illumination, multiple scatterings, etc.). The problem originates from energy received by a detector which is reflected from the object point of interest to other regions of the scene prior to being received by the detector.
The problem is inherent in all triangulation based systems because the necessary extended IFOV along the position sensing axis allows for integration of secondary reflections. It is not uncommon for the secondary reflected beam to contain a strong specular component exceeding the signal obtained from the point of interest, thereby invalidating a typical naive assumption that the largest intensity reading corresponds to the point of interest.
It is important to note that the 3D imaging problem, particularly for triangulation based imaging, can become very complex when imaging curved, specular surfaces which also produce partial occlusion. As taught by Barrow and Tennenbaum noted above, the stray light associated with the occluded regions can be stronger or competitive with primary returns from other points of interest.
Steep slopes and small radii of curvature which are on the order of the system resolution lead to extreme requirements for electronic and optical dynamic range. Perhaps a worst case scenario to consider is thread gaging of a shiny, fine pitch screw with viewpoint of the triangulation receiver constrained as shown in FIG. 1. Such a scenario requires very wide electronic and optical dynamic range.
Prior Art Solutions for Reducing Imaging Errors
Intra-Scene Reflections
Spatial filtering methods taught in the art have been reasonably successful for reduction of secondary reflection components along the scan direction (i.e. substantially orthogonal to the position sensing dimension).
U.S. Pat. No. 4,553,844 (Nakagawa et al.) describes filtering of strong multiple reflections which corrupt 3D data for measurement of solder joints. The IFOV along the scan direction is limited to a narrow strip within an object scanned system in which a "descanning" method is used. There is no discussion, however, about suppression of errors along a position sensing axis.
U.S. Pat. No. 4,643,578 (Stern) teaches the use of a programmable mask in a detector plane synchronized with motion of a laser spot to reject strong background energy. Unwanted energy resulted from welding are glare and ambient light. This method allows for synchronization with standard t.v. sensors and eliminates moving parts.
U.S. Pat. No. 4,634,879 (Penney) and U.S. Pat. No. 4,645,917 (Penney et al.) describe the use of a small mask to reject background light in a similar laser scanning/descanning system, performing a similar function.
"Synchronous scanning" techniques as described in these references have inherent spatial filtering capability.
However, the required IFOV substantially along the position sensing direction often allows for much stronger secondary reflection components to be received, particularly for systems having extended depth of field requirements where surfaces are located in close proximity. In restricted application scenarios, like microelectronics inspection, reduction in this error can be achieved with appropriate adjustments of the viewpoint and the use of multiple sources and detectors and knowledge based data processing techniques. Examples are shown in the above co-pending applications, U.S. Pat. No. 4,529,316 (DiMatteo), U.S. Pat. No. 5,118,192 (Chen et al.), U.S. Pat. No. 4,891,772 (Case et al.) and a commercial IPK system manufactured by Panasonic Inc.
In U.S. Pat. Nos. 5,024,529 and 4,796,997 (Svetkoff et al.), the use of a mask, preferably one which is programmable, is taught in which the IFOV along the height (position sensing dimension) is correlated with the height profile of the object to be inspected. If the mask is used with appropriate polarizers, the polarization delivered to the detector can be selected. Inherent in the laser scanning (transmitter) system is a beam of linear or circular polarized light incident upon the object (produced by the combination of a laser and AO deflector). The use of polarization discrimination in conjunction with the mask was suggested in the '529 patent for eliminating errors associated with shiny objects like metallic interconnects in electronics industry: wire bonds, pin grid arrays, reflowed solder, etc.
Another method to suppress or filter multiple reflections is to orient the projection and imaging subsystems so as to be symmetric about the normal to the surfaces to be imaged and utilize matched linear polarizers in the two subsystems. In this method only specular reflections are deemed to be of interest. Secondary and tertiary reflections which remain within the IFOV are of necessity; the product of at least one diffuse reflection. Since diffuse reflections are known to partially de-polarize incident light, these reflections are suppressed by the linear polarizer in the second subsystem. This technique is not useful for objects which have complex, sharply varying, or concave surfaces, however, as the symmetric orientation condition will generally not be met unless the orientation of the projector and the receiver are adjusted.
In U.S. Pat. No. 5,028,138 (Wolff), a polarization-based method for material classification is disclosed in which specular and diffuse components resulting from multiple reflections can be distinguished. The method is based upon a receiver oriented to receive specular reflected light. A multiplicity of received polarization components are analyzed.
PCT published application WO 94/09463 discloses an apparatus for detecting ice or snow on a surface including a linear polarizer and a wave retarder plate which charges linearly polarized light into circularly polarized light.
Recent polarization-based research work disclosed by Clark et. al. in their articles entitled "Polarization Based Peak Detection in Triangulation Range Sensors" and "Improving Laser Triangulation Sensors Using Polarization," 0-8186-7042-8/95 IEEE also demonstrates specific improvements in triangulation based imagery with the use of polarization discrimination as suggested in the aforementioned earlier work. A LCD camera is preferred to analyze three polarization component images (typically with 0.degree., 45.degree., and 90.degree. analyzer orientations) to reduce or eliminate the ambiguity associated with the important case of metal--metal inter-reflections. The cited work in progress will be extended to dielectric and translucent surfaces.
Instrument-Based Back-Reflections
Yet another problem can arise to further corrupt the measurements. Back reflection from the instrument onto the scene can modify the intensity distribution, particularly when multiple detectors are used (which are intended for the purpose of reducing reflection artifacts in addition to eliminating occlusion). It is known in the illumination and spectral measurement art that wedged windows are useful for eliminating second surface reflections which normally occur with parallel windows, and such reflections corrupt spectral measurements. An increase in the wedge angle provides for increased reflected beam angular separation at a distance. Although not widely documented, in commercial fiber optic systems, back-reflection introduces non-uniformity in the delivered illumination and thermal effects at the source which is sometimes reduced by tilting the light source with respect to the fiber by several degrees.
"Self imaging" effects called "narcissism" are considered in imaging system design (i.e., see, for example, W. L. Wolfe, "Imaging Systems," Infrared Handbook Infrared Information and Analysis Center, 1978, Chapter 19, pp. 19-22,19-23). A single detector views a constant field and stray radiation is negligible. The back reflection problem just described is different in that 1) a second detector receives a weak component of reflected light from the point of interest; 2) the object is sequentially illuminated; and 3) the second detector receives a significant component of stray reflected energy. In the reference cited immediately above, it is noted that spurious reflections can cause crosstalk with neighboring elements of a detector array, the reflections originating from other surfaces within the optical system.
Back-Reflection Effects on Laser Beam
Furthermore, systems utilizing lasers for scanning a beam or projecting a line can be degraded by "feedback light." Specular (or semi-specular) back reflection from the scene may induce destructive interference in the laser and induce improper laser operation. In effect, the scene, in combination with optical surfaces, acts like an external cavity mirror. In prior art systems, optical isolators were used to eliminate this undesirable phenomena.